Development of Short Test Forms within Item Response Theory: The shortIRT Package

Livio Finos\(^{1,2}\), Ottavia M. Epifania\(^{2, 3}\), Pasquale Anselmi\(^{1}\), Egidio Robusto\(^{1}\)

\(^1\) University of Padova, Padova \(^2\) Psicostat, Padova \(^3\) University of Trento, Rovereto

Short test forms

Short Test Forms – Why?


Many items \(\rightarrow\) good measurement precision, great reliability and so on

Not always!


People might get tired and frustrated

IRT models for the win

Being focused on the item information and on the ability of each item to measure different levels of the latent trait, IRT models provide an ideal framework for developing STF (and not torturing people)

Different approaches: Automated test assembly procedures

Item Response Theory

Item Response Function

According to the 4-Parameter Logistic Model:

\[P(x_{pi}=1|\theta_p, b_i, a_i, c_i, e_i) = P(\theta) = c_i + (e_i-c_i)\dfrac{\exp[a_i(\theta_p - b_i)]}{1 + \exp[a_i(\theta_p - b_i)]}\]

\(b_1 = b_2 = 0\), \(a_1 = a_2 = 1.2\)

\(c_3 = c_4 = 0\), \(e_3 = e_4 = 1\)

Item Characteristics Curves (ICCs)

Information Functions

\[ \text{IIF}_{i}(\theta) = \dfrac{a_i^2[P(\theta)-c_i]^2[e_i - P(\theta)]^2}{(e_{i}-c_i)^2 P(\theta)[1-P(\theta)]}\]


\[TIF(\theta) = \sum_{i = 1}^{|B|} IIF_i(\theta)\] (\(B\): Set of items in a test (\(|X|\) cardinality of set \(X\)))

Development of short test forms (STFs)

Benchmark Procedure

Create a short test form composed of \(N\) items from an item bank \(B\) \(\rightarrow\) Select the \(N\) items with the highest IIFs:

The IIFs of the items of item bank are sorted in decreasing order:

\[\mathit{iif} = (\displaystyle \max_{1 < i < B} IIF_i(\theta), \ldots \displaystyle, \min_{1 < i < B} IIF_i(\theta)) \]

Items with IIFs from 1 to \(N\), \(N < ||B||\), are selected to be included in the short test form

Aim: STF with \(N = 3\) items from a full-length test of 10 items:

Item \( b_i \) \( a_i \) \( c_i \) \( e_i \) \( \text{IIF}_i(\theta) \)
1 1.65 1.32 0.10 1 0.36
2 -1.82 0.71 0.06 1 0.11
3 2.87 0.78 0.00 1 0.15
4 -1.79 0.81 0.01 1 0.16
5 -0.83 0.87 0.08 1 0.16
6 1.46 1.35 0.03 1 0.43
7 2.87 0.73 0.00 1 0.13
8 -0.01 1.41 0.06 1 0.44
9 -2.92 1.09 0.06 1 0.26
10 -1.44 1.07 0.09 1 0.24

Aim: STF with \(N = 3\) items from a full-length test of 10 items:

Item \( b_i \) \( a_i \) \( c_i \) \( e_i \) \( \text{IIF}_i(\theta) \)
8 -0.01 1.41 0.06 1 0.44
6 1.46 1.35 0.03 1 0.43
1 1.65 1.32 0.10 1 0.36
9 -2.92 1.09 0.06 1 0.26
10 -1.44 1.07 0.09 1 0.24
4 -1.79 0.81 0.01 1 0.16
5 -0.83 0.87 0.08 1 0.16
3 2.87 0.78 0.00 1 0.15
7 2.87 0.73 0.00 1 0.13
2 -1.82 0.71 0.06 1 0.11

Aim: STF with \(N = 3\) items from a full-length test of 10 items:

Item \( b_i \) \( a_i \) \( c_i \) \( e_i \) \( \text{IIF}_i(\theta) \)
8 -0.01 1.41 0.06 1 0.44
6 1.46 1.35 0.03 1 0.43
1 1.65 1.32 0.10 1 0.36
9 -2.92 1.09 0.06 1 0.26
10 -1.44 1.07 0.09 1 0.24
4 -1.79 0.81 0.01 1 0.16
5 -0.83 0.87 0.08 1 0.16
3 2.87 0.78 0.00 1 0.15
7 2.87 0.73 0.00 1 0.13
2 -1.82 0.71 0.06 1 0.11

\(\theta\)-target procedure

\(k = 0, \ldots, K\): Scalar denoting the iterations of the procedures (\(K = N-1\))

\(S^k \subseteq \{1, \ldots, J\}\): Set of items selected to be included in the short test form up to iteration \(k\)

\(Q^k \subseteq \{1, \ldots, N\}\): Set of \(\theta'\)s satisfied up to iteration \(k\);

At \(k=0\): \(S^0 = \emptyset\), \(Q^0 = \emptyset\)

The procedure cycles steps 1 to 3 until \(k = K\):

  1. Select \(iif_{in}^k = \displaystyle \max_{i \in B\setminus S^k, \, n \in N \setminus Q^k} \mathbf{IIF}(i,n)\);
  2. Compute \(S^{k+1} = S^k \cup \{i\}\) as the set of item selected at \(k\);
  3. Compute \(Q^{k+1} = Q^k \cup \{n\}\) as the set of \(\theta'\)s satisfied at \(k\);

At iteration \(K\), \(|Q^{K + 1}| = N\) and \(|S^{K + 1}| = N\)

\(\theta'\)
1 2 \(\ldots\) n \(\ldots\) N
1 \(\mathit{iif}_{11}\) \(iif_{12}\) \(\vdots\)
2 \(\mathit{iif}_{21}\) \(\mathit{iif}_{22}\) \(\vdots\)
\(\vdots\) \(\vdots\)
\(i\) \(\ldots\) \(\ldots\) \(\ldots\) \(\mathit{iif}_{in}\) \(\ldots\) \(\ldots\) \(\ldots\)
\(\vdots\) \(\vdots\)
\(B\) \(\vdots\) \(\mathit{iif}_{BN}\)

Aim: Develop a STF of \(N=3\) items from \(B\) with \(\theta' = (-2,0,2)\):

Item bank B
Item b a c e
1 1.65 1.32 0.10 1
2 -1.82 0.71 0.06 1
3 2.87 0.78 0.00 1
4 -1.79 0.81 0.01 1
5 -0.83 0.87 0.08 1
6 1.46 1.35 0.03 1
7 2.87 0.73 0.00 1
8 -0.01 1.41 0.06 1
9 -2.92 1.09 0.06 1
10 -1.44 1.07 0.09 1

IIF Matrix \(k = 0\)
0.00 0.08 0.35
0.11 0.08 0.03
0.01 0.05 0.13
0.16 0.10 0.03
0.11 0.15 0.05
0.00 0.16 0.38
0.01 0.05 0.12
0.05 0.44 0.10
0.21 0.04 0.01
0.21 0.15 0.02

\(S^0 = \emptyset\)

\(Q^0 = \emptyset\)

IIF Matrix \(k = 0\)
0 0.08 0.35
0.11 0.08 0.03
0.01 0.05 0.13
0.16 0.1 0.03
0.11 0.15 0.05
0 0.16 0.38
0.01 0.05 0.12
0.05 0.44 0.1
0.21 0.04 0.01
0.21 0.15 0.02

\(\mathit{iif}_{\text{max}}^0=\displaystyle \max_{j \in J\setminus S^0, \, n \in N \setminus Q^0} \mathbf{IIF}= \mathbf{IIF}(8,2) = 0.44\)

\(S^{1} = S^0 \cup \{8\}\) = {8}

\(Q^{1} = Q^0 \cup \{3\}\) = {2}

IIF Matrix \(k = 1\)
0 0.08 0.35
0.11 0.08 0.03
0.01 0.05 0.13
0.16 0.1 0.03
0.11 0.15 0.05
0 0.16 0.38
0.01 0.05 0.12
0.05 0.44 0.1
0.21 0.04 0.01
0.21 0.15 0.02

\(\mathit{iif}_{max}^1=\displaystyle \max_{j \in J\setminus S^1, \, n \in N \setminus Q^1} \mathbf{IIF} = \mathbf{IIF}(6,3)= 0.38\)

\(S^{2} = S^1 \cup \{6\} = \{8, 6\}\)

\(Q^{2} = Q^1 \cup \{3\} = \{2, 3\}\)

IIF Matrix \(k = 2\)
0 0.08 0.35
0.11 0.08 0.03
0.01 0.05 0.13
0.16 0.1 0.03
0.11 0.15 0.05
0 0.16 0.38
0.01 0.05 0.12
0.05 0.44 0.1
0.21 0.04 0.01
0.21 0.15 0.02

\(\mathit{iif}_{max}^2=\displaystyle \max_{j \in J\setminus S^1, \, n \in N \setminus Q^1} \mathbf{IIF} = \mathbf{IIF}(9,1)= 0.21\)

\(S^{3} = S^2 \cup \{9\} = \{8, 6, 9\}\)

\(Q^{3} = Q^2 \cup \{1\} = \{2,3, 1\}\)

End
0 0.08 0.35
0.11 0.08 0.03
0.01 0.05 0.13
0.16 0.1 0.03
0.11 0.15 0.05
0 0.16 0.38
0.01 0.05 0.12
0.05 0.44 0.1
0.21 0.04 0.01
0.21 0.15 0.02


\(|S^3| = 3\), \(|Q^3| = 3\), \(K = 2\) \(\rightarrow\) end

The shortIRT package

It’s on CRAN!

install.packages("shortIRT")
library(shortIRT)


bench()

bench(item_par, iifs = NULL, theta = NULL, num_item = NULL)
set.seed(1312)
n = 10 
item_par = data.frame(b = runif(n, -3,3),
                      a = runif(n, .7, 1.5),
                      c = runif(n, 0, .10),
                      e = 1)
theta = rnorm(1000)
stf = bench(item_par, theta = theta, num_item = 3)
1
Define the item parameters in the item bank
2
Random values for the latent trait
3
Generate the STF with the benchmark procedure

bench()

bench(item_par, iifs = NULL, theta = NULL, num_item = NULL)
set.seed(1312)
n = 10 
item_par = data.frame(b = runif(n, -3,3),
                      a = runif(n, .7, 1.5),
                      c = runif(n, 0, .10),
                      e = 1)
theta = rnorm(1000)
stf = bench(item_par, theta = theta, num_item = 3)
summary(stf)
plot(stf) + theme_bw()
1
Define the item parameters in the item bank
2
Random values for the latent trait
3
Generate the STF with the benchmark procedure
4
Summary of the obtained STF
5
Plot the resulting TIF (as compared to the TIF obatined from \(B\)) 
summary(stf)         
The selected items are
10 2 9 
These items maximize the information for thetas equal to: 
2.251 -1.215 -1.077 
 The parameters of the selected items are: 
           b        a          c e
10  2.243866 1.390367 0.02130947 1
2  -1.246596 1.283032 0.02087452 1
9  -1.171959 1.344295 0.07825710 1
The item selection is based on the bench procedure 
plot(stf)

define_targets()

define_targets(theta, num_targets = NULL, method = c("equal", "clusters"))

targetsC = define_targets(theta, 
                          num_targets = 3, 
                          method = "clusters")
targetsC
          1           2           3 
-1.20701883  1.21315753  0.01562251 
attr(,"class")
[1] "clusters"

targetsE = define_targets(theta, 
                          num_targets = 3, 
                          method = "equal")
targetsE
[1]  0.045  2.485 -2.395
attr(,"class")
[1] "equal"

theta_target()

theta_target(targets, item_par)
stfC = theta_target(targetsC, item_par)
summary(stfC)
The selected items are
2 6 8 
These items maximize the information for thetas equal to: 
-1.207019 1.213158 0.01562251 
 with the following parameters 
           b        a          c e
2 -1.2465959 1.283032 0.02087452 1
6  0.8283301 1.198013 0.01028904 1
8  0.3331977 1.208366 0.09370018 1
The item selection is based on the theta-target procedure with clusters target
plot(stfC, show_both = F)

stfE = theta_target(targetsE, item_par)
summary(stfE)
The selected items are
10 8 2 
These items maximize the information for thetas equal to: 
2.485 0.045 -2.395 
 with the following parameters 
            b        a          c e
10  2.2438664 1.390367 0.02130947 1
8   0.3331977 1.208366 0.09370018 1
2  -1.2465959 1.283032 0.02087452 1
The item selection is based on the theta-target procedure with equal target
plot(stfE, show_both = F)

User defined targets

The selected items are
10 6 4 
These items maximize the information for thetas equal to: 
2 2 2 
 with the following parameters 
           b         a          c e
10 2.2438664 1.3903672 0.02130947 1
6  0.8283301 1.1980125 0.01028904 1
4  0.7957328 0.9924159 0.08716286 1
The item selection is based on the theta-target procedure with unknown target

Final remarks

Boh funziona si possono dire tante cose ma non so cosa dire